An Evaluation of Inter-Annotator Agreement in the Observation of Anaphoric and Referential Relations

When proposing a description of the data he observes, the linguist must make sure that his observations may be also regularly made by other persons. In this paper, we introduce a typology of anaphoric and referential relations and an experiment which aims at assessing that this typology is operational. Given three newspaper articles, five students were asked to identify anaphoric and/or referential relations between expressions and referents. This inter-subjectivity test confirms results already obtained: coreference is an operational notion, but the perspicuity of other relations is not obvious. Linguists make use of many different notions which often overlap. It is the case for the notions of “coreference” and “anaphora” in the domain we are concerned with, namely text interpretation. Coreference, understood as the identity of reference of distinct expressions, includes cases of anaphora, but not all cases of anaphora (we have met the terms “associative anaphora”, “indirect anaphora”, “one anaphora”, among others; none were used to refer to cases of coreference strictly speaking). Things get more complicated when coreference, for some, may extend beyond strict identity. To get a grasp of this problem, we defined a typology of anaphoric and referential relations, which is presented in the first part of this paper. The goal of this paper, however, is not the presentation of this typology per se. Through an experiment aiming at assessing that our typology of anaphoric and referential relations was operational, in the sense that different persons could use it to make the same observations, we would like to show that the sophistication of linguistic descriptions may find its limits in the testing of their operationality. In the perspective of a natural language processing system, it is important to select phenomena for which there is an operational description, in particular as such a description is necessary to evaluate the system. The first part of the paper presents some preliminary notions and our typology of anaphoric and referential relations (sections 1 and 2). Section 3 introduces the experiment which aims at assessing the operationality of our typology. Evaluation criteria for the different types of observations to be made are given in section 4, then evaluation measures in section 5. The next sections introduce the results (section 6), organized with respect to the different types of relations to be observed, and a discussion of these results (section 7). 1. Preliminary notions Denotation world. We take for granted that to a text is associated what we call a “denotation world”, the world 1 It is the case in (Chinchor and Sundheim, 1995). denoted by the text, and we say that expressions in this text “describe” this denotation world. Basically, the denotation worlds which we assume are associated to texts are “the vast complex of things and situations that sentences can be about” (Dowty et al., 1981). A given text may eventually describe several denotation worlds. Referents. We assume the denotation world associated to a text is populated with “beings” or “referents”: the objects, persons, events, facts, situations, etc. the text talks about. The sentence John loves Mary, for instance, describes a denotation world which contains three referents: the one denoted by John, the one denoted by Mary and the one denoted by the whole sentence, i.e. the fact that John loves Mary. Referential and non-referential expressions. In a given text, some expressions have the particularity to point or refer to referents of the denotation world while others do not. We say that the former are “referential” and the latter “non-referential”. In the sentence This shirt is blue, we consider that This shirt is a referential expression; it points to an object in the denotation world associated to the sentence. The expressions blue, or is blue, or even shirt alone, on the other hand, are nonreferential; they do not point to a referent but only describe the referent pointed to by This shirt. Anaphora. As a rule, expressions in a text are to be interpreted in a context. The expression the president of the United States, for instance, may denote Bill Clinton in one context, or Georges Washington in another, or Franklin Roosevelt in yet another context, etc. Similarly, the proper name Bill Clinton may be used to refer to the man who is at present president of the United States, but it might very well be used to refer to some other man 2 A given text may describe a denotation world which may or may not be the real world. In both cases, we will talk about “referents”; our referents are discourse referents (cf. (Karttunen, 1976)). 3 By “context”, without further qualification, we always mean any kind of context, text or situation. References to a specific kind of context will be made clear by appropriate description (e.g. “textual context”, “non-textual context”). bearing this name. The precise reference of expressions, while being usually unique in a given context, varies from context to context. One note, however, that different expressions impose different constraints on the way they may possibly be interpreted, depending on the richness of their descriptive content. For instance, we would say that the descriptive content of the president of the United States is richer than that of the president, which is richer than that of he. Any man referred to by the first expression may also be referred to by the two others; any man referred to by the second expression may also be referred to by the third; but the inverse of these two statements is not true. We call “anaphoric expressions” expressions which have some incomplete descriptive content and as such are interpreted in relation to some other contextual expressions, i.e. a textual context. More specifically, we will consider as possible anaphoric expressions pronominal noun phrases, possessive determiners, temporal adverbial expressions, or noun phrases determined by a definite article or a demonstrative. In the following sentence, 1. Comme elle l'avait laissé entendre en février, la BNP a décidé de rapprocher ses filiales de crédits spécialisés. [As it let it be understood in February, the BNP has decided to merge its subsidiaries of specialized leasing.] we consider that the personal pronoun elle and the possessive determiner ses are anaphoric with respect to la BNP, that the clitic pronoun l' is anaphoric with respect to the clause la BNP a décidé de rapprocher ses filiales de crédits spécialisés. Allowing some extension of the notion of anaphora to non-textual contexts in the case of temporal expressions, we will also consider that the phrase en février is anaphoric in that it does not provide a complete description of its referent, in this case the month of February 1998, and the identification of this referent relies on contextual information, in this case the date of the article this example has been taken from. We will use the term “anaphora” both for cases where the contextual expression precedes or follows the anaphoric expression. 2. A typology of anaphoric and referential relations Taking as a starting point the interpretation of anaphoric expressions, we propose a typology of the different relations between expressions which we think play a role in the interpretation of texts, in particular in the interpretation of noun phrases. We are here concerned with relations between expressions insofar as these relations are not expressed syntactically. The term “expression” here is also to be considered in a broad sense; it includes expressions proper (the words which are actually in the text), but also “elliptical expressions”, which may be inferred from the resolution of ellipsis, ellipsis being induced by some expressions proper. We distinguish two main types of relations depending on whether the relation involves a relation between the 4 This presentation of the notion of anaphora is partly inspired from (Ranta, 1994). referents of the two expressions, or whether it only involves their description. The former are called “referential relations”, the latter “description relations”. 2.1. Referential relations In referential relations, both related expressions are referential. In our typology, five types of relations are considered: coreference, distinction, set-membership, part-of and a fifth underspecified relation for relations which are not any of the four other types. 2.1.1. Coreference If two referential expressions denote the same referent, the relation is coreference. In the following sentence, 2. Allianz présente son nouveau visage. [Allianz introduces its new face.] the expressions Allianz and son refer to the same referent; they are coreferential. An expression ei may corefer with a set of expressions ej, ..., en if it denotes a set S and the expressions ej, ..., en each denote a distinct element or subset of S and the full extension of S is specified by ej, ... en. In the sentence: 3. Allianz rappelle que ses 10% d'Ergo ne sont pas stratégiques et que les deux compagnies restent des concurrentes acharnées. [Allianz insists that its 10% of Ergo are not strategic and that the two companies remain fierce competitors.] the expression les deux compagnies denotes a set composed of two entities, the referent of Allianz and the referent of Ergo. We consider that les deux compagnies is coreferent with the set composed of the two expressions Allianz and Ergo. 2.1.2. Distinction The relation called “distinction” chiefly aims at giving an account of the anaphoric relation in the use of the adjective autre (“other”). The adjective autre may be seen as a two place predicate: if some referent oi is described as autre, it is so in relation with another referent oj. We say that oi is distinguished from oj. In the following sentence, 4. Cette annexe parisienne du Palais de justice [...] devrait rapidement être suivie d'autres pôles en province. [This Parisian annex of the Law Courts should be soon followed by other poles in the